Existing prediction systems utilize information associated with agents in an environment to infer future trajectories of the agents. However, existing prediction systems are reliant on accurate agent classification (e.g., pedestrian, cyclist, vehicle, etc.) and/or on a naïve assumption of agent dynamics. Inaccurate classification, or the inability to classify an agent (e.g., due to ambiguity and/or insufficient data), can introduce prediction errors.